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Assessing the safety effects of prediction errors and uncertainty on automation- 
supported functions in the Next Generation Air Transportation System concept of 
operations is of foremost importance, particularly safety critical functions such as separation 
that involve human decision-making. Both ground-based and airborne, the automation of 
separation functions must be designed to account for, and mitigate the impact of, 
information uncertainty and varying human response. This paper describes an experiment 
that addresses the potential impact of operator delay when interacting with separation 
support systems. In this study, we evaluated an airborne separation capability operated by a 
simulated pilot. The experimental runs are part of the Safety Performance of Airborne 
Separation (SPAS) experiment suite that examines the safety implications of prediction 
errors and system uncertainties on airborne separation assistance systems. Pilot actions 
required by the airborne separation automation to resolve traffic conflicts were delayed 
within a wide range, varying from five to 240 seconds while a percentage of randomly 
selected pilots were programmed to completely miss the conflict alerts and therefore take no 
action. Results indicate that the strategic ASAS functions exercised in the experiment can 
sustain pilot response delays of up to 90 seconds and more, depending on the traffic density. 
However, when pilots or operators fail to respond to conflict alerts the safety effects are 
substantial, particularly at higher traffic densities. 
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I. Introduction 

A SSESSING the safety effects of human response characteristics on automation-supported functions in the Next 
Generation Air Transportation System (NextGen) concept of operations 1 is of foremost importance, 
particularly safety critical functions, such as separation, that involve human decision-making. The automation of 
separation functions, both ground-based and airborne, must be designed to account for, and mitigate the impact of, 
varying human response. Delayed responses to conflict alerts may result in late resolutions possibly leading to losses 
of separation. Automation systems for separation require operators to interact in a timely manner responding to 
predicted hazardous conditions in accordance with procedures and training. This paper describes an experiment that 
addresses the potential impact of operator delay when interacting with separation support systems within the context 
of the aircraft crew interacting with an Airborne Separation Assistance Systems (ASAS). By understanding the 
impact of delayed pilot responses to conflict alerts we can establish design guidelines for crew procedures and 
automation alert timing in ASAS applications. Similar studies need to be performed for all automation-enabled 
separation assurance applications that rely on the human operators to preserve safe separation. 

This study is part of the Safety Performance of Airborne Separation (SPAS) experiment 2 * * suite that comprises a 
series of batch simulation studies investigating the safety impact of prediction errors and system uncertainties on 
ASAS applications. Current SPAS experiments are based on scenarios consisting of randomized routes in a generic 
high-density airspace in which all aircraft are constrained to the same flight level. Sustained average traffic density 
is varied from 1 1.2 to 21.4 aircraft per 10,000 square nmi, approximating up to about 10 times today’s traffic density 
in a typical sector. During the progression of experiments, various errors, uncertainties, delays, and other variables 
potentially impacting system safety will be incrementally introduced to analyze the effect on safety of the individual 
factors as well as their interaction and collective effect. Results from the baseline SPAS study 2 indicate that at five 
times the typical traffic density of today’s National Airspace System (NAS) and under the assumptions of the study, 
airborne separation can be safely performed. 

In the current study, pilot actions required by the ASAS automation to resolve traffic conflicts were varied over a 
wide range of response times, varying from 5 to 240 seconds, while a percentage of randomly selected pilots were 
programmed to completely miss the conflict alerts and therefore take no action. 

The paper is organized as follows: Section II presents a brief summary of previous work, Section III describes 
the simulation platform, and Sections IV and V describe the experiment design and results. Finally, Section VI 
presents conclusions and future direction research. 


II. Background 

The addition of automation to support the separation function is a response to the capacity limiting effect of 
human-based separation control of today’s system 3,4 . Automation can remove workload bottlenecks, improve 
prediction of conflicts as traffic patterns become more dense and complex, and provide conflict-free maneuvers 
alternatives to human operators. Integrated air/ground operational concepts have been proposed in which some 
aircraft crews exercise separation functions aided by ASAS tools on the flight deck, while air traffic controllers 
exercise ground based separation control for non- AS AS-equipped aircraft and terminal operations 5 . ASAS systems 
integrate advanced decision-aiding automation into aircraft avionics. These decision aids rely on data-linked, 
broadcast surveillance information that includes aircraft velocity vectors and limited flight plan information through 
a surveillance capability such as the Automatic Dependent Surveillance - Broadcast (ADS-B) 6 . The automation is 
designed to detect conflicts between aircraft and generate conflict resolution routes and conflict-free maneuvering 
advisories 7,8,9 . 

Until now, safety evaluation of SA applications has, for the most part, been based on studies using simulation 
tools that seldom include models of system uncertainties 10,11,12 and often make many simplifying assumptions such 
as perfect navigation performance and absence of prediction errors or off-nominal conditions. Analytical and 
numerical approaches to the analysis of safety using uncertainty propagation models often require many simplifying 
assumptions to keep the size of the models manageable or computation time within reasonable limits 13 . Monte Carlo 
simulations are frequently an alternative when the analysis requires higher fidelity, although simplifying 
assumptions are often required to more quickly achieve large amounts of data. However, statistically valid 
predictions are elusive when based on the occurrence of events that are seldom observed such as the critical 
separation violations in normal air traffic conditions. For that reason, special techniques for the modeling of rare 
event probabilities are being explored in the context of Air Traffic Management (ATM) operations 14 . Although 
these studies are key to understanding the behavior of large complex systems such as the NextGen, it is important to 
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complement them with high fidelity, detailed studies that incorporate models of uncertainty sources, measure 
performance in the presence of prediction errors and operator performance variability. 

The SPAS simulation study attempts to characterize and quantify the safety performance of ASAS applications 
using a simulation platform that includes high fidelity models of aircraft dynamics, flight management system, data 
link communications, and conflict detection and resolution functions as described in the next section. The baseline 
set of runs for the SPAS experiment, completed in the spring of 2007, included scenarios with no system 
uncertainties or prediction errors 2 . Results from that study indicated that within the experimental conditions and 
assumptions, safety was preserved with no losses of separation observed for traffic densities much higher than 
current levels. For the baseline SPAS test runs described in the aforementioned study 2 , ADS-B reception was perfect 
(i.e., all messages received at all ranges), full aircraft trajectory intent was shared, the pilot responded correctly to all 
traffic alerts with no delays, and wind predictions were accurate. Results showed that at five times the typical traffic 
density of today’s NAS, utilizing only airborne strategic conflict detection and resolution logic with a 10-minute 
look-ahead time, safe separation of aircraft can be maintained under the assumptions and conditions of the test. 
Subsequent studies underway and planned seek to identify the conditions and estimate the traffic levels that could 
compromise safety in the presence of various sources of uncertainty. 

One of the problems in conducting a pilot response delay study on airborne separation is the lack of prior 
experimental or measured data describing valid pilot response delay ranges in similar circumstances or for similar 
SA systems. These applications are context sensitive and human performance is influenced by specific training and 
procedures as well as other required cockpit tasks. As indicated in a report on testing and designing new cockpit 
alerting systems 15 , there are new applications and procedures requiring alerting systems that “self-activate in time- 
critical, hazardous situations”. Multiple factors can affect pilot response time and pilot conformance, such as lack of 
trust as a result of frequent occurrences of false alerts or conflicting training instructions. The operational procedure 
within which the alerting system is used is likely to affect the operator’s response time. If pilots are trained and 
expected to react within a certain time period and are given responsibility for separation assurance, the response 
time would be required to be within a bounded time range. In some cases, the alerting system is part of a procedure 
that requires pilot compliance with immediate actions within a given time, such as the TCAS II procedure in which 
pilots are expected to act within 5 seconds of an alert. While there is a lack of studies and operational experience on 
which to base the requirements for pilot response times for airborne separation it is possible to investigate the 
effects of delayed pilot responses on safety using ad-hoc time delay ranges to measure the performance of relevant 
safety metrics. For this study, pilot response time ranges were configured to reach levels that could compromise 
separation in order to observe potential safety degradation. Additional experiment outcomes include procedure 
design guidelines for required pilot response times for future systems. 

III. The Simulation Platform 
A. The Airspace & Traffic Operations Simulation 

The SPAS experiments are being conducted at the Air Traffic Operations Laboratory (ATOL) at NASA Langley 
Research Center utilizing a distributed simulation platform that includes a network of aircraft simulators 
interconnected through a 
High Level Architecture 
(HLA) communication 

infrastructure. The simulation 
platform, known as the 
Airspace & Traffic 
Operations Simulation 

(ATOS) 16 can be used for 
batch Monte Carlo studies as 
well as real-time human-in 
the-loop experiments. The 
ATOS is comprised of 
hundreds of real-time, high 
fidelity aircraft simulators 
equipped with experimental 
displays and pilot interfaces. 

Each aircraft simulator is referred to as the Aircraft Simulation for Traffic Operations Research (ASTOR). 
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Conceptually, the batch simulation platform is a multi-agent system composed of multiple distributed autonomous 
software agents modeling AS AS equipped aircraft flown by a rule-based pilot model as shown in Fig. 1. Each 
aircraft agent has its own view of the airspace, and there is no centralized system controlling or modifying the paths 
of the aircraft or behavior of the pilot models. 

Each ASTOR includes a six degree-of-freedom dynamic airplane model, a Flight Management System (FMS), 
Mode-S ADS-B datalink capability, and a prototype AS AS called the Autonomous Operations Planner (AOP) that is 
described in the next section. For batch simulations, all ASTORS are “flown” by a rule-based pilot model (PM) that 
performs the basic pilot actions for airborne conflict management as described below. 

B. Airborne Conflict Management System 

The airborne conflict management system used in this experiment series is the Autonomous Operations Planner 
(AOP), a NASA-developed research model of an airborne automation system built for the study of advanced 
distributed air-ground operational concepts 7 . The AOP provides an integrated suite of capabilities for managing 
traffic conflicts and trajectory changes from the flight deck perspective, including conflict detection, resolution, 
prevention, and trajectory constraint conformance. These capabilities are developed to a level of fidelity sufficient 
for research of complete airborne responsibility for self- separation as defined by the Principles of Operation of 
ASAS Category 4, “Airborne Self-Separation” 17 . In the JPDO NextGen Concept of Operations, they correspond to 
the functions of Separation Management and aspects of Trajectory Management 1 . 

The intent-based conflict detection (CD) function of AOP uses state and intent data received from other traffic 
aircraft over ADS-B in combination with ownship state data, auto-flight mode settings, and flight plan information 
to deterministically predict future losses of separation. AOP also has a second, independent, CD system that uses 
state-vector projections to detect flight crew blunders and prediction faults of the intent-based CD system and other 
short time horizon conflicts. Conflict alerting is modeled after the multi-alert-level approach recommended by 
RTCA 18 . The timing of alerts between aircraft are staggered as a method for incorporating a right-of-way rule set 
(i.e., priority rules), based roughly on the set used in Visual Flight Rules 19 . In the current study, the simple 
computer-based pilot model reacted to alerts within a normally distributed timing scheme defined by the parameters 
of the experiment, described later. When alerted to a conflict, the pilot model’s action was to request a fully-conflict- 
free resolution trajectory from AOP. 

For conflict resolution (CR), AOP contains both strategic and tactical capabilities. Tactical CR refers to open- 
loop vectors or altitude changes to solve conflicts with no predetermined reconnection plan. A Strategic CR refers 
to the single action of modifying the flight plan such that the conflict is solved and the aircraft reconnects to the 
previous trajectory. In generating solutions, AOP’s strategic CR system takes into account all known trajectory 
constraints, including trajectories of nearby traffic, airspace hazards, 
ownship performance limits, and required time of arrival (RTA) 
constraints. A genetic algorithm is employed to search within a set 
of pre-defmed geometric patterns to generate CR trajectories that 
simultaneously accommodate these constraints 20 . Once a suitable 
CR trajectory is calculated, and it is verified conflict free for a 
nominal time of 20 minutes based on current traffic data, it is 
presented to the flight crew for review. Immediately upon execution 
of the route, ADS-B broadcasts the new ownship intent as a series of 
trajectory change points. 

AOP also includes functions for conflict prevention, including 
at-a-glance maneuver restriction symbology for the flight crew and 
support for tactical / strategic trajectory probing (also known as 
provisional CD). These functions were not required for this study 
and were therefore disabled. 

C. The Pilot Model 

Pilot model agents are self-contained agents capable of making 
independent decisions, and taking actions based upon their 
perceived state of the traffic and airspace. The PM reactive rules- 
based logic was designed to interact with the conflict management 
system by responding to conflict alerts with the action of accepting a 

conflict resolution advisory. Fi § ure 2: Pi,ot Model Architecture 
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The PM design is composed of three parts: a sensory input model, a rule-based decision model, and an actuator 
response model as shown in Fig. 2. The sensory input model gathers information by detecting changes in the AOP 
interface such as a new conflict alert. A sensory time delay was implemented to model cases when a pilot does not 
notice a conflict alert immediately. This delay can be individually configured for each action that the PM is designed 
to perform. For the strategic conflict detection and resolution interface that is exercised in this experiment, there are 
three different pilot actions required and they are described in more detail in Section IV. The rules module 
determines the appropriate action and commands the actuator module to execute it. The actuator model simulates 
the pilot action required to execute a command, such as pushing a button, and its responses are also governed by a 
time delay algorithm. 


IV. Experiment Description 
A. Experiment Scenarios and Assumptions 

A simulated en route airspace modeled for the SPAS experiment is depicted in Fig. 3. The test region, a notional 
airspace en route sector, is modeled as a circular area with a diameter of 160 nmi. The test region is surrounded by 
an initialization region, the outer boundary of which is the location where aircraft are initialized in the simulation. 
This initialization 
method provides 
each aircraft’s AOP 
with a full 10- 
minute look-ahead 
time for detecting 
conflicts that occur 
within the test 
region. Aircraft are 
generated at random 
points on the outer 
circle initially with 
straight trajectories 
that traverse the test 
region at random 

angles and continue to a waypoint with a required time of arrival. All aircraft in the simulation are ASAS equipped 
and fly at the same altitude so as to constrain the scenarios to lateral conflict resolutions only and to achieve higher 
traffic densities. More details on the SPAS scenario design can be found in the Preliminary SPAS experiment 
report 2 . 

For this study, the auto-flight system remained coupled to the FMS for lateral navigation such that there was no 
tactical maneuvering. No altitude changes were permitted. There were no wind prediction errors or ADS-B message 
degradation due to signal range or interference. The AOP was configured to exercise only the intent-based CD and 
strategic CR capabilities with a look-ahead time of 10 minutes and a separation minimum of 5.1 nautical miles. 
ADS-B communications included full intent data. A priority rule (i.e., right of way) system was in effect that 
prevented undesirable synchronicity of resolution maneuvers by both aircraft involved in a conflict. Aircraft given 
priority for a given conflict had their alerts delayed 3 minutes while the other conflicting aircraft were alerted 
immediately. In Fig. 3 aircraft B has an initial trajectory that is in conflict with aircraft A, just entering the 
initialization zone. Aircraft B resolves the conflict by modifying its fly path as shown in the diagram. 

The state-based CD and the tactical CR capability were disabled to allow the current study to focus on strategic 
conflict management. The strategic resolution logic attempts to find an RTA-conforming route that is conflict-free 
for 20 minutes. Nominal look-ahead time is 10 minutes, but the strategic CR will attempt resolutions with as little 
as 2 minutes to separation loss. If a resolution is found, it is guaranteed to preserve separation in the absence of 
prediction errors, even if only one aircraft executes it. If a strategic resolution cannot be found in time, the system 
would normally transition to a tactical resolution phase (typically with three minutes to go), which is not present in 
this experiment. The result of not having the tactical back-up system is that conflicts irresolvable by the strategic 
system of both aircraft will result in separation loss. Nevertheless, all losses of separation (LOS) must be accounted 
for regardless of their severity. Future studies will incorporate both the strategic and tactical capabilities. 
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B. The Pilot Model Configuration 

For this experiment the PM was designed to interact with the human interface currently implemented for the 
ASTOR’s strategic conflict detection and resolution functionality. This interface requires three pilot actions: First, 
when a conflict is detected and its alert is shown on the navigation display, the PM must request AOP to compute a 
resolution trajectory. This action represents the pilot’s first awareness of a potential conflict and it is modeled by a 
delay shown in Fig. 4 as dl. The PM reaction to the conflict alert is to request AOP to compute a resolution route. 
Once computed, a resolution route is shown to the pilot on the experimental navigation display. 

The time elapsed 
between the PM 
request and when the 
resolution route is 
completed is a 
dependent metric 
referred to as the 
Resolution Cycle 
Time. This is the time 
it takes AOP to 
compute a conflict 
free trajectory based 
on all the known 
traffic information. 

This time is strongly 
affected by traffic 
density and in this 

current study ranged between mean values of 5 to 25 seconds for traffic densities ranging from 1 1.4 to 20.3 average 
aircraft per 10000 nmi2. Second, when the new route is displayed, a pilot would have the opportunity to review the 
new trajectory before uploading it. This is modeled by the PM as the d2 delay shown in Fig. 4. When the resolution 
route is uploaded, it becomes a MOD route for the FMS and the pilot is required to execute it before it becomes the 
active route. The time before this last PM action is modeled with a time delay shown as d3 in Fig. 4. The traffic 
conflict is then removed from the display. All PM delays for this experiment were modeled as normally distributed 
random variables with mean values ranging from 3.5 to 240 seconds. The lowest delay value is used to represent a 
“typical” pilot response delay. The responsiveness factor represents pilots that ignore conflicts alerts and do not 
attempt to resolve them. The responsiveness factor is modeled as a uniformly distributed random variable 
representing the percentage of times that all pilots ignore conflicts. This variable is sampled for each individual 
conflict, therefore, it is possible for two pilots involved in a conflict to both ignore it and never request a resolution. 
In the current study, the pilot model was programmed to execute the first available CR trajectory with an average 
delay of five seconds from the time the resolution route is shown. 
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Figure 4. Conflict Alert Timeline 


C. Experiment Design 

A three-factor, response surface experiment design was used to explore the effect of average traffic density, pilot 
response delay and pilot 
non-responsiveness, on 
safety. The two safety 
metrics of interest in this 
experiment are losses of 
separation (LOS) and the 
time remaining before LOS 
at resolution execution. 

The experiment consisted 
of 85 simulation runs in 
which the three factors 
were assigned values in the 
ranges shown in Table 1. 

Table 2 shows the relationship of mean traffic density levels tested to current-day levels, as well as the number 
of flight hours of data collected and conflicts recorded. In total, 6978.5 flight hours of data were collected, and 
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Table 1: Experiment Design 


Factors 

Units 

Min 

Max 

Traffic Density 

Number of Aircraft per 
10000 nmi 2 

11.2 

21.4 

Pilot Delay 

Seconds 

3.5 

240 

Pilot Non- 
Responsiveness 

Percentage of pilots not 
responding to conflict 
alerts 

0 

10 


24384 conflicts were generated. As expected, conflicts per flight hour increased with traffic density. It should be 
noted that this metric reflects all 
traffic in the airspace, not conflicts 
per single aircraft flight hour. The 
approximate mean and peak ratios 
are based on an analysis of traffic 
demand levels in the NAS that 
estimated the traffic count for every 
high altitude sector in the United 
States, at each flight level, for the 24 
hour period of 19 February 2004 
(ref. SPAS baseline). From this 
data, the mean and maximum traffic count for FL3 1 0 (the busiest altitude within the selected sector) was determined 
for the median-density sector ZOA31. The traffic levels were then normalized to produce densities based on 
10,000 nmi 2 . The ranges used in this study represent an estimated five to twelve times the average traffic density of 
the median sector and three to eight times its peak density. 

V. Experiment Results 


Table 2: Summary of Data Collection Runs 


Average 
Density 
Per 10000 
nmi 2 

Approximate 
T raffic Ratio 
(mean/peak) 

Flight 

Hours 

Total 

Conflicts 

Conflicts / 
Flight 
Hour 

11.2 

5X/3X 

1375.7 

3273 

2.38 

16.3 

8X/6X 

1918.4 

6361 

3.32 

21.4 

12X/8X 

3684.4 

14750 

4.00 


A. Impact of Traffic Density 

The effect of average traffic density with normal pilot delays and all pilots responding can be observed in Table 
3. Preliminary analysis indicates that at the highest density level and under the constrained conditions of the 
experiment scenarios, the 
resolution function was able to 
find a strategic (lateral) solution 
in all but two cases. In fact, out 
of 2307 conflicts generated in 
572 simulated flight hours there 
were 2 instances where a 
resolution was not found. The 
case involved a 4-aircraft conflict 
in which one of them lost 
separation with two of the intruders. Further analysis is required to identify the cause of the 2 resolution failures 
observed. As it was determined in earlier studies 2 , at traffic densities above 17 aircraft per 10,000 nmi 2 , the three 
resolution patterns used in computing strategic resolutions appear to provide insufficient maneuver flexibility in a 
few observed cases. It is hypothesized that the addition of maneuver patterns adds degrees of freedom and would 
provide the needed flexibility. The full implementation of AOP also uses a tactical resolution algorithm to handle 
exception cases in which a full strategic solution cannot be found before three minutes to go. This tactical capability 
was not exercised in this study. Also, adding a vertical degree of freedom to the resolution computation is likely to 
further improve the performance. 

B. Impact of Pilot Delay on LOS 

The effect of pilot delay with all pilots responding indicates a non-decreasing trend as both density and delay 
increase (Table 4). For that reason, it is important to perform an in-depth analysis of the results to identify the causes 
of the intrusions observed. Figure 5 depicts a notional timeline diagram of the conflict detection and resolution cycle 
in the presence of pilot delays from the viewpoint of the two aircraft involved. This diagram shows how, in some 
situations, it is possible that sufficient time may not be available to solve a conflict, thereby leading to an eventual 
LOS. In the diagram, Aircraft 1 and 2 are in conflict with each other. The conflict detection time horizon used in the 
study was 10 minutes, and therefore the alert for Aircraft 1 is shown with approximately 600 seconds remaining 
before LOS. The blue (solid) distribution indicates the range and probability of pilot delay, an input parameter for 
the simulation defined by a mean and standard deviation. The long mean delay condition of 240 seconds is shown. 
After some randomly selected pilot delay within this distribution, the pilot model requests a resolution and the 
resolution cycle begins. The green (dashed) distribution indicates the range of resolution cycle performance over a 
wide range of conflict geometries and traffic conditions. Depending on the specific traffic situation, a resolution 
trajectory may be easy (A) or hard (B) to compute, which results in an additional delay before the conflict is solved. 
If the situation is too complex, a resolution trajectory for Aircraft A may not be forthcoming. 
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Table 3: Impact of Traffic Density 


Average 

Density 

Average 

Pilot 

Delays 

Flight 

Hours 

Sum 

Conflicts 

Total 

LOS 

LOS/ 

Conflicts 

11.2 

3.5 

240.73 

583 

0 

0 

16.3 

3.5 

90.71 

316 

1 

0.003164 

21.4 

3.5 

572.76 

2307 

2 

0.000866 





Meanwhile, the alert for Aircraft 2 is offset by three minutes to prevent synchronous conflict resolutions with 
Aircraft 1. Following this offset, the conflict alert is given and the pilot delay begins. As the figure shows, a three 
minute offset followed by a roughly four minute pilot delay leaves only about three minutes to solve the conflict. 
Strategic solutions may be more difficult to compute in this short time frame because maneuver options are limited; 
this can be seen by the notional difference in the resolution cycle performance distribution as compared to 
Aircraft 1. If the traffic situation is such that a solution is found (C), then the conflict is resolved (if not already 
resolved by Aircraft A). On the other hand, a strategic solution may be elusive in this time frame (D). In AOP, at 
one minute to LOS, the strategic CR 
algorithm is disabled. The current 
design for AOP is to switch to a 
tactical CR system at three minutes 
(180 seconds) to LOS. This tactical 
system was not tested in the current 
study. Therefore, at short time frames 
resulting from excessive pilot delays, 
the ability to solve conflicts 
strategically comes down to 
resolution cycle performance. 

Resolution cycle performance for a 
given algorithm design and 
implementation is generally a 
function of three factors: the traffic 
situation (including geometry and 
density), the time remaining before 
LOS, and the degrees of freedom 
available for generating possible 
solutions. Reductions of one or more 
of these parameters may have the 
effect of extending the time required 
to find a conflict-free solution, and in 
some cases, may prevent such solutions altogether. Such cases would show up as LOS events provided that no other 
resolution system is present. 

Of these factors in the current study, the traffic geometry was randomly generated, and the traffic density and 
pilot delay were parameters of the simulation. The remaining factor, degrees of freedom, was determined by the 
experiment setup and algorithm 
design. The experiment setup 
permitted no vertical solutions, which 
in itself significantly limited the 
degrees of freedom. In the lateral 
dimension, AOP searches three 
solution patterns: lateral offsets, path 
stretches, and direct intercepts 20 . 

These three patterns have been found 
to be largely adequate for solving 
randomly generated conflicts with full 
look-ahead time up to traffic densities 
of ~17 aircraft per 10,000 nmi 2 . (See 
future work.) The detailed analysis on 
the causes of the observed LOS is 
fundamental to the design and 
optimization of CD&R automation 
and it can only be done with high Figure 6: Number of multi-Aircraft Conflicts in a Single Run. 
fidelity models. One of the conditions 

that seems to affect the conflict resolution performance is the number of multi-aircraft conflicts encountered by the 
resolution logic. 
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Figure 6 shows the distribution of multi-aircraft conflicts observed during three scenario runs corresponding to a 
pilot delay of 240 seconds. The graph shows the number of conflicts binned in terms of the number of conflicting 
aircraft pairs, for example three aircraft pairs indicate four aircraft involved (ownship plus three intruders). Note 
that the involved aircraft are not necessarily all in conflict with each other. 

As Table 4 indicates, at the lowest traffic density of 11.2 aircraft per 10,000 nmi 2 , there was a single intrusion 
observed during the 240 seconds delay scenario. The intrusion was determined to be a multi-aircraft conflict. At the 
middle density, there were 5 LOS, 3 of them were minimal (distance at the point of closest approach was greater 


Table 4: Experiment Data Runs per Density and Pilot Delay 


Aircraft 

per 

10000 

nmi 2 

Average 

Pilot 

Delay 

(sec) 

Total 

Flight 

Hours 

Total 

Conflicts 

Total 

LOS 

LOS/ 

Conflicts 

LOS / Flight 
Hour 

11.2 

(5X/3X) 

3.5 

240.73 

583 

0 

0 

0.000 

46.0 

59.21 

150 

0 

0 

0.000 

90.0 

300.38 

744 

0 

0 

0.000 

240.0 

59.52 

142 

1 

0.0070 

0.017 

16.3 

(8X/6X) 

3.5 

90.71 

316 

1 

0.0031 

0.011 

46.0 

89.01 

289 

1 

0.0034 

0.011 

90.0 

87.03 

284 

1 

0.0035 

0.011 

240.0 

88.33 

296 

2 

0.0067 

0.023 

21.4 

(12X/8X) 

3.5 

572.56 

2307 

2 

0.0008 

0.003 

46.0 

115.13 

484 

1 

0.0020 

0.009 

90.0 

577.48 

2463 

7 

0.0028 

0.012 

240.0 

117.35 

514 

10 

0.0194 

0.085 


than 4.92 nmi) the other two were multi-aircraft conflicts. At a density of 21.4 aircraft, there were ten LOS and nine 
of them involved multi-aircraft conflicts. The preceding analysis of the causes for the observed intrusions suggests 
that the complexity of the conflicts encountered and the delayed pilot response, both have an effect on the LOS 
metric. 

C. Impact of Delay and Density on the Resolution to LOS time 

The resolution execution time relative 
to the time of the predicted first loss of 
separation (FLOS) was found to be 
sensitive to pilot delay as expected. The 
detailed description of the cause for this 
response is provided in the preceding 
section, in Figure 4 and 5. This time is 
inversely related to pilot delay, reaching 
the lowest observed value in the 
experiment for pilot delays of 240 seconds 
as it can be seen in the graph in Fig. 7. 

The effect of delayed resolutions on LOS 
begins to be observed in the experiment at 
240 seconds. It is at that time that AOP 
strategic resolutions may not have time to 
converge to a solution prior to the 3- 
minute-to-LOS limit. This metric shows 
the resilience of the strategic CR to long pilot delays. 


Average Times to First LOS 



PM Request Delay 

Figure 7: Average Time from Resolution to FLOS 
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D. Impact of Pilot Responsiveness on LOS 

The effect of pilot responsiveness and traffic density for all combined pilot delays can be observed in Table 5. 
When a pilot of one aircraft is non-responsive to a conflict alert, the onus is placed on the pilot of the other aircraft 
to resolve the conflict. As shown in Fig. 5, when one aircraft does not respond, the other aircraft still faces the same 
challenges in coming up with a strategic resolution in time. A significant delay caused either by the pilot or by the 
various factors that impact resolution cycle performance, discussed earlier, can result in no solution being found in 
sufficient time, therefore leading to a LOS. If the non-responsive pilot is flying Aircraft 1, then the priority-based 
alerting offset adds an additional delay which further increases the likelihood of a LOS. Of course, if both pilots are 
non-responsive, then a separation loss is assured. By multiplying the probability of non-responsiveness for any 
given aircraft (e.g., 10%) by that of any aircraft it may be in conflict with (e.g., 10%), the number of conflicts that 
result in a LOS due to this factor can be estimated (e.g. 1% of conflicts). As Table 5 indicates, the percentage of 
LOS at the highest densities and lowest responsiveness levels is twice as much for the delays combined, (e.g., 2%). 
In other words, the LOS shown are a combination of all the delay conditions. 


Table 5: Data Runs shown by Pilot Responsiveness and Traffic Density 


Non- 

Resp. 

Density 

Flight 

Hours 

Total 

Conflicts 

Total 

LOS 

LOS/ 

Conflicts 


11.2 

659.84 

1619 

1 

0.0006 

0 

16.3 

511.93 

1711 

7 

0.0041 


21.4 

1498.1 

5799 

37 

0.0064 


11.2 

0 

0 

0 

0 

98 

16.3 

332.6 

1121 

7 

0.0062 


21.4 

457.15 

1940 

16 

0.0082 


11.2 

60.38 

149 

2 

0.0134 

95 

16.3 

706.31 

2297 

18 

0.0078 


21.4 

350.77 

1432 

19 

0.0133 


11.2 

655.49 

1505 

41 

0.0272 

90 

16.3 

266.39 

887 

10 

0.0113 


21.4 

1378.4 

5652 

114 

0.0202 


VI. Conclusion and Future Work 

This study is part of an on-going series of high fidelity batch simulation experiments seeking to understand the 
safety impact of increased traffic density and system uncertainties. This paper studies the effects of human operator 
inattentiveness when interacting with cockpit based automated systems used for separation assistance. These effects 
were observed by varying pilot delays and responsiveness within wide ranges. LOS count and resolution to FLOS 
time were evaluated to assess the performance of the system under the experimental conditions. An in-depth analysis 
of the underlying causes for the observed behavior revealed great stability of the airborne strategic resolution 
capability under a large range of conditions. Results from this study can contribute to the design requirements for 
procedures and applications involving automated separation assurance systems. These results begin to reveal some 
of the traffic and system conditions that if not addressed could compromise safety such as operator response delay in 
very dense traffic. Finally, understanding the safety effects of human operator performance variability will enable 
the design of preventive procedures and more robust automation. Future experiments incorporating the tactical AOP 
capability as well as vertical resolution and possible optimized resolution patterns are being planned. Also, studies 
of other sources of uncertainties such as wind prediction errors and datalink communication degradation as well as 
mixed equipage operations are underway. 
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